On the Consistency of Feature Selection using Greedy Least Squares Regression

This paper studies the feature selection problem using a greedy least squares regression algorithm. We show that under a certain irrepresentable condition on the design matrix (but independent of the sparse target), the greedy algorithm can select features consistently when the sample size approaches in nity. The condition is identical to a corresponding condition for Lasso. Moreover, under a sparse eigenvalue condition, the greedy algorithm can reliably identify features as long as each nonzero coe cient is larger than a constant times the noise level. In comparison, Lasso may require the coe cients to be larger than O( √ s) times the noise level in the worst case, where s is the number of nonzero coe cients.